fix(tbench2_env): a missing reward.txt is a scoring error, not reward 0.0 - #1025
Merged
nblintao merged 2 commits intoAug 29, 2026
Merged
Conversation
… 0.0 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1012, closing the last gap where an infrastructure failure could masquerade as a task failure (spotted by @Shi-Dong reviewing the downstream consumer).
What
When the canonical harness runs but no verdict is ever written —
tests/test.shcrashed or was killed before its verifier wrote/logs/verifier/reward.txt(e.g. the uvx toolchain download failed, or the in-shelltimeoutfired) —evaluatepreviously returned reward 0.0 with the normal canonical-harness info, byte-for-byte indistinguishable from tests genuinely failing.Now
_parse_canonical_rewardreturnsNonefor an absent/empty/non-numeric verdict and both callers (local + docker mode, via a shared_require_canonical_verdict) raise:step()reports it through its existing error path —observation.errorset,reward=None, episode keeps its session — with the test.sh log tail in the message for diagnosis. Verify-window cleanup still runs (covered by tests in both modes).Why it matters
RL consumers must be able to drop a no-verdict episode: ingesting it as 0.0 injects a false negative into training that no client-side check can catch (the reply looked completely valid). A genuine test failure still scores 0.0 — test.sh's verifier writes
reward.txt = 0in that case, so the only replies that now error are ones where the verifier never ran to completion.Behavior change
Only the no-verdict edge case:
evaluatereplies that used to be a silent0.0are now an explicit scoring error. Genuine passes/failures and the pytest-fallback path (which scores by exit code and always has one) are unchanged.Validation
pytest tests/envs/test_tbench2_env.py→ 26 passed: two new tests (local + docker) assert an empty reward marker raisesRuntimeError("produced no verdict")and that staged verifier assets are still wiped on the error path.🤖 Generated with Claude Code
Note
Medium Risk
Changes only the no-verdict edge case for canonical harness scoring, but it alters
evaluateobservability for RL consumers and could surface previously silent infrastructure failures as errors.Overview
When the canonical
tests/test.shharness runs but never writes a verdict (reward.txtmissing, empty marker, or non-numeric value), scoring no longer returns 0.0 as if tests failed._parse_canonical_rewardnow returnsNonein those cases instead of coercing to 0.0._require_canonical_verdictraisesRuntimeErrorwith a log tail so local and Docker canonical evaluate paths surface a scoring failure throughstep()(observation.error,reward=None) rather than a false negative for RL pipelines.Genuine pass/fail (numeric
reward.txt) and the pytest fallback path are unchanged. New unit tests cover the empty-marker case in both local and Docker modes and assert verifier staging still cleans up on error.Reviewed by Cursor Bugbot for commit 84bcc90. Bugbot is set up for automated code reviews on this repo. Configure here.